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ABSTRACT 

Simulation  has  overcome  critical  obstacles  to  become  a  valuable  method  for  obtaining  insights  about  the 
behavior  of  complex  systems.  George  Box’s  famous  assessment  that  “all  models  are  wrong,  some  are 
useful”  referred  to  statistical  models,  but  should  now  be  reimagined  to  reflect  that  many  simulation  mod¬ 
els  are  “right  enough”  to  aid  in  decision  making  for  important  practical  problems.  Over  the  past  fifty 
years,  simulation  has  transformed  from  its  beginnings  as  a  brute-force  numerical  integration  method  into 
an  attractive  and  sophisticated  option  for  decision  makers.  This  is  due,  in  part,  to  the  exponential  growth 
of  computing  power.  Although  other  analytic  approaches  also  benefit  from  this  trend,  keyword  searches 
of  several  scholarly  search  engines  reveal  that  the  reliance  on  simulation  is  increasing  more  rapidly.  A  de¬ 
scriptive  analysis  paints  a  compelling  picture:  simulation  is  frequently  a  researcher’s  preferred  method 
for  supporting  decision  makers  and  may  often  be  the  “first  resort”  for  complex  real  world  issues. 

1  INTRODUCTION 

The  1958  article,  “Simulation  Techniques  in  Operations  Research”  by  John  Harling,  described  simulation 
as  a  tool  of  “last  resort.”  Harling  considered  simulation  to  be  analogous  to  waving  the  white  flag  and  set¬ 
tling  for  an  approximate  solution  when  it  becomes  too  difficult  to  break  a  system  down  into  a  “simple” 
model.  However,  three  sentences  after  Harling  called  simulation  a  method  of  last  resort,  he  went  on  to 
mention  that  that  the  cost  of  simulation  often  outweighs  the  benefits  because  “large  machines  are  so  often 
involved.”  This  criticism  seems  laughable  today,  because  Harling  had  yet  to  realize  the  phenomenon  that 
Gordon  Moore  would  formalize  in  1965,  when  he  predicted  the  exponential  growth  of  computing  power. 
While  simulation  was  once  an  expensive  and  cumbersome  brute-force  means  of  estimating  an  answer, 
today’s  analysts  have  more  than  10  billion  times  more  processing  power  (according  to  Moore’s  Law)  than 
Harling  did.  With  this  expanded  processing  power,  as  well  as  advances  in  the  practice  of  simulation,  it 
makes  sense  that  simulation  should  be  an  increasingly  valuable  technique. 

Astonishingly,  the  argument  that  simulation  should  be  a  last  resort  is,  unfortunately,  alive  and  well 
fifty  years  later.  In  their  2008  article,  “Dynamic  Allocation  of  Airline  Check-In  Counters:  A  Queueing 
Optimization  Approach,”  Parlar  and  Sharafali  (2008)  constructed  a  complex  queuing  model  to  “optimize” 
the  number  of  employees  that  should  man  the  counters  for  a  modem  airline.  Aside  from  a  number  of 
oversimplified,  unrealistic  assumptions  (such  as  exponential  service  times,  independence,  and  a  known 
number  of  only  one  customer  at  a  time  arrivals)  that  allowed  for  tidy  math,  the  article  claimed  that  the 
problem  should  be  tackled  via  “analytical  treatment”  rather  than  “resorting”  to  simulation  because  that 
approach  is  “more  realistic”  and  the  “results  are  on  firmer  ground.” 

In  this  paper,  we  present  the  results  of  keyword  analysis  involving  several  large  search  engines.  This 
assessment  of  simulation  publications  provides  one  measure  of  how  the  field  has  developed.  We  find  that 
the  usage  of  simulation  has  grown  exponentially,  much  like  the  computing  power  that  supports  such 
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models.  A  comparison  of  search  engine  searches  for  analytical  keywords  “linear  programming”  and  “op¬ 
timization”  shows  that  the  growth  of  these  particular  topics  has  not  kept  pace  with  that  of  simulation.  A 
more  detailed  look  at  influential  years,  and  historical  developments  leading  up  to  these  years,  provides 
more  context  about  the  way  simulation  has  developed  as  a  field.  We  feel  that  understanding  this  context 
is  an  interesting  and  useful  topic  for  both  undergraduate  and  graduate  courses,  as  we  prepare  the  next 
generation  of  simulation  modelers,  analysts,  and  researchers.  Moreover,  teachers  can  use  these  findings 
to  motivate  their  students,  as  they  show  the  value  of  the  material  being  taught. 

2  EARLY  ROOTS 

Simulation  proved  useful  as  an  analytic  technique  long  before  the  astonishing  growth  in  computing  pow¬ 
er.  In  order  to  fully  understand  and  appreciate  the  impressive  rise  in  the  use  of  simulation,  one  must  pay 
respect  to  the  genesis  of  its  expansion.  In  the  early  computer  days  of  the  1950s  and  1960s,  simulation 
was  by  no  means  a  new  concept.  The  use  of  simulation  predates  computers,  with  perhaps  the  most  fa¬ 
mous  example  being  Buffon’s  “needle  experiment”  to  estimate  the  value  of  n  in  1777  (Nance  and  Sargent 
2002).  Another  well-known  simulation  took  place  in  1908,  when  William  Gosset  used  manual  simulation 
to  empirically  derive  an  estimate  of  the  probability  density  function  for  his  “Student’s  ‘t’  distribution 
(Student  1908).  Over  the  years  researchers  were  beginning  to  adopt  the  opinion  that  simulation  could  be 
used  as  a  means  of  putting  years  of  research  and  data  to  work.  Similarly,  the  use  of  simulation  motivated 
further  research  and  stimulated  advancement  across  all  operational  fields  as  well  as  the  business  and  mili¬ 
tary  worlds. 

In  the  business  world,  the  dollar  is  the  bottom  line.  With  that  philosophy  in  mind,  it  makes  sense  that 
simulation  began  to  gain  popularity  in  the  business  world  as  the  cost  of  sophisticated  computers  began 
falling.  In  1954,  Cuthbert  C.  Hurd  wrote  of  a  machine-shop  scheduling  problem  that  was  addressed  using 
simulation  after  other  methods  failed.  Researchers  from  the  1953  Operations  Research  Society  of  Ameri¬ 
ca  meeting  attempted  to  solve  the  scheduling  problem  through  analytical  means.  Specifically,  they  tried 
to  apply  “the  theory  of  game  and  of  linear  programming,”  but  found  those  methods  to  be  of  little  help. 
What  was  of  great  help  was  the  IBM  701,  a  machine  that  “calculated  directly”  the  solution  to  the  ma¬ 
chine-scheduling  problem  through  simulation  (Hurd  1954). 

Some  of  Hurd’s  words  were  unique  in  that  he  complemented  mathematicians  and  their  deep  explora¬ 
tion  of  complex  systems  and  the  formulations  that  motivate  them.  He  credited  their  studies  for  benefitting 
the  physical  world  and  for  igniting  the  desire  for  further  research.  Hurd’s  tip  of  the  hat  to  theoretical 
mathematicians  is  in  stark  contrast  to  the  common  simulation  versus  analytical  solution  arguments  that 
exist  among  some  scholars.  Hurd  suggested  that  “simulation  by  computation”  was  the  key  to  unearthing 
deeper  mysteries.  He  went  on  to  mention  that  simulation  was  continuing  to  gain  popularity  thanks  to  the 
falling  “unit  cost  of  carrying  out  computation.”  In  other  words,  computers  were  getting  cheaper.  Think¬ 
ing  back  to  the  “bottom  line”  mentality,  a  cheap  machine  that  computed  the  most  cost-effective  decision 
rules  made  perfect  sense  as  the  weapon  of  choice  for  high-level  decision  makers  (Hurd  1954). 

Speaking  of  weapons,  let  us  recognize  the  early  uses  of  simulation  as  a  tool  in  tactical  decision  mak¬ 
ing.  That  tool  is  known  as  the  “war  game.”  In  a  1955  paper  titled  “Simulation  as  an  Aid  in  Model  Build¬ 
ing,”  R.  P.  Rich  discussed  a  model  that  was  based  on  a  fleet  air  defense  system.  He  stressed  that  the  fact 
that  a  military  example  was  being  used  was  purely  “coincidental,”  because  almost  any  complex  scenario 
could  have  been  used  to  demonstrate  the  utility  of  simulation.  Rich’s  paper  is  valuable  in  that  it  illustrates 
many  benefits  that  arise  from  simulation,  but  his  words  on  war  gaming  are  particularly  relevant  in  the 
way  they  highlight  how  simulation  has  been  used  to  model  battlefield  operations.  In  a  peculiar  way,  Rich 
(1955)  used  the  word  “machine”  to  describe  simulation  before  he  or  the  world  had  fully  come  to  realize 
the  impact  that  machines  would  have  on  the  subject.  When  he  wrote  that,  “a  war  game  is  just  a  special 
type  of  simulator,  with  the  opposing  teams  so  many  parts  of  the  machine,”  he  was  not  referring  to  a  com¬ 
puter  that  ran  the  model,  but  rather  the  individuals  whose  actions  provided  the  decisions  within  the  sce¬ 
nario  and  added  stochastic  variability.  The  people  playing  the  game  were  key  moving  parts.  Rich  did, 
however,  mention  a  “simple  device”  that  played  a  role  in  “emulation”  for  the  fleet  air  defense  scenario.  It 


Powers,  Sanchez,  and  Lucas 


was  the  combination  of  the  device  and  the  opposing  team  “parts”  that  provided  the  basis  for  this  very  spe¬ 
cial  type  of  simulator.  The  war  game  would  prove  to  be  a  type  of  model  that  would  play  an  important 
role  in  strategic  and  tactical  development  across  all  branches  of  the  military. 

Today,  the  United  States  Department  of  Defense  (DoD)  regularly  utilizes  hundreds  of  large  simula¬ 
tions  (M&SCO  2012)  to  help  in  deciding  how  forces  should  be  equipped,  organized,  trained,  employed, 
deployed,  and  maintained.  To  capture  the  complexities  of  the  real  world,  these  simulations  sometimes 
involve  many  thousands  of  input  variables  and  can  take  several  hours  to  run  a  single  Monte  Carlo  exper¬ 
iment.  One  such  simulation  environment  in  widespread  use  is  the  Extended  Air  Defense  Simulation 
(Teledyne  Brown  Engineering  2009).  First  deployed  in  1989,  EADSIM  was  used  to  analyze  air  opera¬ 
tions  in  advance  of  and  during  operation  Desert  Storm.  EADSIM  is  currently  being  used  by  nearly  400 
agencies  around  the  world.  Its  success  derives  in  large  part  from  its  object  oriented  design  and  pre  and 
post  processing  graphical  interfaces.  As  with  most  DoD  models,  EADSIM  has  followed  one  of  this  pa¬ 
per’s  author’s  “Lucas’s  Law,”  that  is:  “The  detail  within  a  model  grows  in  proportion  to  the  processing 
power  available,  leaving  runtimes  relatively  constant.” 

3  SEARCH  ENGINE  KEYWORD  ANALYSIS 

To  investigate  the  behavior  of  the  growth  of  simulation,  and  compare  it  to  a  few  other  fields,  we  use  key¬ 
word  searches  for  several  scholarly  search  engines.  Through  data  collection  via  varying  search  engines, 
exploration  of  the  increase  in  use  of  simulation  was  not  limited  to  the  scope  of  specific  journals.  The 
year-by-year  keyword,  title,  and  abstract  searches  by  journal  display  increases  in  the  frequency  and  pro¬ 
portion  of  simulation-inspired  articles.  Each  search  engine  allows  for  an  “advanced  search”  that  permits 
the  filtering  of  articles,  year-by-year,  in  such  a  way  that  only  those  articles  with  the  central  topic  of  simu¬ 
lation  appear  on  the  results  page.  The  search  engines  provided  by  the  Naval  Postgraduate  School  library 
online  database  used  for  this  research  are  JSTOR  (Journal  Storage),  INFORMS,  EBSCOhost  (Elton  B. 
Stephens  Co.),  and  ACM  DL  (Association  for  Computing  Machinery  Digital  Library).  These  search  en¬ 
gines  also  display  the  relevant  article  counts  per  year,  so  data  collection  became  a  simple  (yet  painstak¬ 
ing)  matter  of  recording  the  results  in  a  form  amenable  for  analysis. 

We  begin  with  simple  graphic  displays  of  the  data,  but  our  analysis  goes  further  by  fitting  regression 
models.  In  these  models,  the  first  year  of  the  collected  data  (1960)  is  “Year  1,”  1961  is  “Year  2”  and  so 
on,  through  “Year  51”  (2010).  The  purpose  of  these  fits  is  not  to  predict  the  future,  but  rather  to  quantify 
past  trends  and  help  identify  years  that  were  particularly  influential  in  the  development  and  acceptance  of 
simulation  as  a  research  tool.  Those  years  that  have  been  identified  as  influential  will  enrich  the  narrative 
and  understanding  of  the  sorts  of  advancements — both  technological  and  theoretical — that  have  had  the 
greatest  impact  on  the  simulation  world.  Years  that  reflect  particularly  important  leaps  (and  drops)  in  the 
frequency  of  simulation  publication  are  also  of  interest.  Due  to  the  lag  between  article  submission  and 
publication,  an  influential  year  actually  represents  a  window  of  time  around  which  to  investigate. 

For  comparison  purposes,  the  “simulation”  curves  appear  alongside  “linear  programming”  and  “opti¬ 
mization”  curves  that,  while  incomplete,  were  collected  in  the  exact  same  matter  as  the  simulation  search¬ 
es.  Our  intent  is  not  to  minimize  the  impact  of  these  important  operations  research  methods,  but  rather  to 
examine  whether  the  increase  in  simulation-based  articles  is  mirrored  across  other  fields  that  benefit  from 
increased  computational  power.  It  is  important  to  note  that  “simulation”  keyword  searches  were  excluded 
from  “optimization”  keyword  searches  to  avoid  overlap  in  results  that  included  articles  with  both  key¬ 
words.  This  allowed  for  articles  that  dealt  with  “simulation  optimization” — a  vibrant  topic  in  the  simula¬ 
tion  community  over  the  past  decade — to  default  as  “simulation”  articles.  For  additional  details  and  dis¬ 
cussion,  see  Powers  (2012). 

3.1  JSTOR  Results 

In  Figure  1,  a  simple  display  of  the  frequency  of  simulation  articles  over  a  fifty-year  time  frame  paints  a 
compelling  picture.  Overall,  the  growth  during  the  first  thirty  years  is  less  than  half  that  of  the  last  twenty 
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years.  The  changes  in  JSTORLP  and  JSTOROpt  are  not  nearly  so  dramatic.  JSTORLP  rises  slightly 
but  then  falls  to  near  its  original  levels,  while  JSTOR  Opt  increases  at  a  slow,  but  steady,  rate. 

The  variability  of  JSTORSim  increases  over  time.  So,  we  transform  the  keyword  search  results 
(JSTOR_Sim)  to  reduce  the  impact  of  heteroscedasticity.  After  fitting  several  regression  models  to  the 
JSTOR  search  engine  data,  we  settle  on  a  final  parsimonious  model  of  the  form: 

JSTOR  Sim0'5  =  5.1870  +  0.4437  *  (Year). 

This  model  has  a  p-value  <  0.0001  and  an  R1  value  of  0.976.  Cook’s  Distance  measures  the  influence  of 
points  on  this  model  form  as  a  combination  of  their  leverage  and  their  residuals.  For  our  final  model,  the 
most  influential  years  are  1960,  1961,  1972,  and  1973.  The  first  two  (1960  and  1961)  will  appear  several 
times  and  will  be  discussed  later  in  this  analysis.  The  remaining  unusual  years  are  1972  and  1973.  Upon 
inspection  of  the  data,  one  can  see  that  there  was  a  particularly  high  jump  in  simulation  articles  between 
1971  and  1972  (120  articles  in  1971  and  171  articles  in  1972).  One  would  expect  to  find  some  interesting 
happenings  that  would  pertain  to  simulation  on  or  about  the  year  1971.  A  simple  internet  search  reveals  a 
few  possibilities  (Computerhope  2012): 

•  1970:  Intel  announced  the  first  RAM  chip,  the  Intel  1103,  with  more  than  1,000  bits  of  memory. 

•  1971:  Intel  developed  the  first  microprocessor,  the  Intel  4004,  capable  of  60,000  instructions  per 
second  and  a  clock  speed  of  740  kHz. 

•  1972:  The  C  programming  language  was  developed.  It  introduced  structured  programming  and 
was  a  prelude  to  object-oriented  programming. 
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Figure  1:  JSTOR  data  scatter  plot.  The  growth  of  the  simulation  data  is  impressive  when  compared  to 
the  slight  decline  of  linear  programming  (LP),  and  the  slow  linear  growth  of  optimization  (Opt)  articles. 


Further  scholarly  research  suggests  some  other  landmarks  that  very  well  may  have  contributed  to  a 
surge  in  simulation  use. 


•  1969:  Alan  Pritsker  of  Purdue  University  produced  GASP  II,  which  is  software  that  aided  in 

making  simulation  programming  languages  portable  across  operating  systems  (Hollocks  2006). 
At  the  time,  this  FORTRAN-based  language  was  praised  for  being  flexible  and  well  document¬ 
ed — and  it  was  already  being  taught  at  Arizona  State  University  (Petersen  1969). 
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•  1970:  B.  W.  Hollocks  implemented  GSP-III’s  features  with  FORTRAN  as  a  platform  to  create 
FORSS  (FORTRAN-based  Simulation  System).  Its  portability  resulted  in  wide  use  (Hollocks 
2006). 

•  1970:  Jeffrey  R.  Raskin’s  “A  Tutorial  on  Random  Number  Generation”  recognizes  the  benefits 
of  computer-generated  pseudorandom  numbers,  particularly  their  ease  of  generation  and  the  abil¬ 
ity  to  reproduce  results  (Raskin  1970).  Raskin’s  paper  represents  one  of  many  at  the  time  that 
dealt  with  the  idea  of  improving  on  existing  methods  of  random  number  generation  because  of  its 
necessity  in  the  field  of  simulation. 

The  analysis  of  the  JSTOR  data  identified  an  influential  window  of  time  from  the  late  1960s  through 
the  early  1970s.  Computational  developments  have  almost  certainly  played  a  role  in  the  apparent  jump  of 
simulation-inspired  articles. 

3.2  INFORMS  Results 

Figure  2  displays  the  results  from  the  INFORMS  search  engine.  Once  again,  we  see  that  the  growth  of 
simulation-related  articles  is  more  dramatic  than  the  growth  of  articles  related  to  linear  programming. 
The  final  model  for  the  INFORMS  search  engine  data  is 

INFORMSSim04  =  3.8179  +  (5.2906  *  Year)  -  (0.8676  *  Year2). 

This  model  has  a  p-value  <  0.0004  for  the  first-order  term  for  Year,  a  p-value  <  0.05  for  the  second-order 
term,  and  an  R2  value  of  0.822.  The  naming  convention  remains  the  same  as  the  JSTOR  model.  The  in¬ 
fluential  years  are  1960,  1963,  and  2008  through  2010.  This  could  simply  because  these  are  a  set  of 
“bookend”  years,  or  because  they  were  truly  unusual.  By  the  year  2008,  advances  in  computing  technol¬ 
ogy  are  so  dense  that  it  is  nearly  impossible  to  pinpoint  several  events  that  could  have  inspired  a  leap. 


Figure  2:  INFORMS  data  scatter  plot.  Once  again,  the  simulation  results  grow  faster  than  linearly  while 
the  LP  results  display  a  linear  trend.  The  search  engine  did  not  allow  for  exclusion  of  “simulation”  from 
“optimization,”  so  no  trend  line  for  “optimization”  exists.  One  of  the  most  interesting  aspects  of  the 
simulation  curve  is  how  the  data  points  drop  below  the  trend  line  for  the  years  2002  -  2007. 

While  the  numerical  results  of  the  INFORMS  search  engine  may  appear  to  be  less  impressive,  the  fact 
that  the  search  results  spawn  from  a  smaller  sample  of  publications  allows  for  a  more  intimate  analysis.  A 
look  at  the  data,  as  represented  on  a  scatterplot,  reveal  an  interesting  story.  An  exploration  into  the 
INFORMS  society  itself  may  explain  the  drop  in  simulation  articles  from  2002  to  2007.  The  following 
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are  summaries  of  key  events  obtained  from  the  minutes  of  the  INFORMS  Simulation  Society  Business 
Meetings  from  1999  through  2008. 

•  May  1999:  At  this  time,  the  organization  was  known  as  the  INFORMS  College  on  Simulation. 
The  College  was  acting  like  a  Society  in  that,  among  other  things,  it  was  running  the  PhD  collo¬ 
quium  at  the  Winter  Simulation  Conference  and  actively  inviting  interested  students  to  attend  the 
conference.  The  minutes  reveal  a  decline  in  College  representation  at  the  annual  meetings  and  a 
need  to  encourage  more  simulation-related  article  submissions. 

•  November  1999:  More  concern  over  the  dropping  number  of  simulation  submissions  was  noted. 
It  is  mentioned  that  the  College  would  have  more  influence  if  it  were  a  society. 

•  October  2004:  The  College  had  over  500  members,  which  was  large  enough  to  be  considered  a 
Society.  The  members  at  the  meeting  unanimously  voted  to  petition  for  Society  status. 

•  November  2005:  The  organization  held  its  first  meeting  as  a  Society.  It  is  mentioned  that  simu¬ 
lation  submissions  are  on  the  rise. 

•  November  2007.  As  a  sign  of  great  success,  it  is  mentioned  that  742  authors  are  contributing  to 
the  Winter  Simulation  Conference,  and  that  all  the  hotel  blocks  reserved  for  the  conference  have 
been  booked. 

By  2008,  the  number  of  simulation-related  articles  had  risen  above  the  trend  line. 


3.3  ACM  DL  Results 

While  the  earlier  results  show  polynomial  growth  in  the  number  of  simulation-related  articles,  the  results 
for  the  ACM  DL  search  engine  data  are  fit  well  by  an  exponential  curve  (p-value  <  0.0001,  R 2  =  0.945). 
Figure  3  shows  the  results  from  the  ACM  DL  search  engine.  The  final  model  is 

log(ACM_Sim)  =  3.4251  +  (0.1051  *  Year). 

The  influential  years  (in  terms  of  Cook’s  Distance)  are  1960,  1964,  1966,  and  1967.  This  set  of  years 
shall  be  discussed  in  the  results  of  the  next  search  engine. 


Figure  3:  ACM  DL  data  scatter  plot.  The  ACM  DL  search  engine  allowed  for  separation  of  “simulation” 
from  “optimization”  searches.  The  “optimization”  (Opt)  and  “linear  programming”  (LP)  curves,  while 
exponential,  are  dwarfed  by  the  “simulation”  (Sim)  curve. 
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A  look  at  the  exponential  curve  and  the  data  point  corresponding  to  the  year  2005  reveals  some  inter¬ 
esting  characteristics.  The  change  from  2000  to  2005  is  larger  than  that  from  2005  to  2010,  while  the 
year  2010  is  below  the  curve.  This  might  suggest  that  not  much  has  happened  between  2005  and  2010  as 
far  as  computing  capability  is  concerned,  and  the  world  may  be  poised  for  another  advance!  The  rather 
large  leap  in  the  plot  between  2000  and  2005  inspires  an  investigation  of  computing  history  as  far  back  as 
2000. 


•  2000:  Intel  releases  the  Pentium  4. 

•  2003:  Intel  releases  Pentium  M. 

•  2005:  Intel  releases  Pentium  D. 

•  2006:  Intel  releases  Core  2. 

•  2007-2009:  Intel  releases  several  advancements  to  the  Core  2. 

These  advancements,  while  rapid  and  significant,  do  not  relate  specifically  to  the  world  of  simulation 
as  to  suggest  that  they  play  the  most  important  role  in  simulation’s  growth.  They  have  certainly  improved 
computers  as  far  as  processing  power  and  overall  speed  are  concerned,  but  perhaps  the  simulation  com¬ 
munity  is  about  to  witness  a  breakthrough  that  is  more  than  a  leap  in  technology. 

3.4  EBSCOhost  Results 

Figure  4  shows  the  results  from  the  EBSCOhost  search  engine.  As  for  the  ACM-DL  search  engine,  the 
growth  is  quite  dramatic.  The  final  model  is 

log(EBSCOhostSim)  =  2.8499  +  0.1 194  *  Year  (p-value  <  0.0001,  R2  =  0.966). 

The  influential  years  (according  to  Cook’s  distance)  are  1961,  1962,  1966  and  1968;  1963  (with  Cook’s 
distance  0.0768)  is  also  very  close  to  the  influential  cutoff  (4 In  =  0.0784).  This  set  of  years  is  remarkable 
in  that  they  represent  early  days  of  computer  simulation  other  than  the  “bookend”  year  of  1960.  This  par¬ 
ticular  analysis  motivates  a  look  at  developments  from  the  early  to  late  1 960s  that  were  influential. 


Figure  4:  EBSCOhost  data  scatter  plot.  The  EBSCOhost  search  engine  allowed  for  separation  of  “simu¬ 
lation”  from  “optimization”  searches.  The  “optimization”  (Opt)  curve,  while  representing  smaller  return 
values,  is  also  exponential. 
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With  1961  through  1963,  1966,  and  1968  identified  as  the  influential  years  for  this  fit,  it  was  neces¬ 
sary  to  look  at  developments  in  the  computing  world  and  the  world  of  academia  as  it  pertained  to  simula¬ 
tion.  With  hindsight  being  what  it  is,  this  necessity  is  because  common  knowledge  tells  us  that  techno¬ 
logical  advances  could  not  have  been  the  only  driving  force  behind  the  advancement  of  simulation 
modeling  in  the  1960s. 

•  1961:  The  programming  language  FORTRAN  IV  is  created. 

•  1963:  IEEE  is  founded.  IEEE  will  be  responsible  for  publishing  many  simulation-based  articles. 

•  1964:  BASIC  is  run  for  the  first  time. 

•  1964:  IBM  introduces  System/360,  which  uses  interchangeable  software.  This  is  an  example  of 
a  “third  generation  computer.” 

•  1965:  Gordon  Moore  publishes  Moore’s  law. 

•  1969:  Lewis,  Goodman,  and  Miller  publish  the  paper,  “A  pseudo-random  number  generator  for 
the  System/360.”  This  paper  describes  “pseudo-random  number  generator  that  uses  the  full  ca¬ 
pacity  of  the  32-bit  registers  of  IBM  SYSTEM/360  computers,”  thereby  addressing  a  hot  topic  of 
the  day  that  was  common  to  critiques  of  simulation. 

The  efforts  of  Peter  A.  W.  Lewis  were  not  the  first  attempt  to  develop  random  numbers  and  variates 
that  behaved  according  to  accepted  statistical  properties.  Such  efforts  dated  as  far  back  as  1 949,  with  the 
multiplicative  congruential  generator  originally  introduced  by  Lehmer.  In  fact,  by  1971,  over  200  papers 
had  been  written  on  the  subject  of  random/pseudorandom  number  generation  (Craddock  and  Farmer 
1971).  This  amount  of  publications  was  indicative  of  the  magnitude  of  attention  that  stochastic  modeling 
was  receiving.  So,  while  he  may  not  have  broken  ground  in  the  random  number  generation  field,  Dr. 
Lewis  was  a  pioneer  in  the  simulation  community  mostly  because  he  was  a  respected  scholar  who  took  it 
seriously. 

As  previously  mentioned,  one  of  the  most  prohibitive  factors  that  had  been  holding  back  the  ad¬ 
vancement  of  simulation  was  the  negative  attitude  that  the  Operational  Research  community  expressed 
towards  it.  Dr.  Lewis  was  one  of  the  first  highly  respected  scholars  to  visualize  simulation  as  a  valid 
method  for  modeling  and  research.  His  endorsement  played  a  tremendous  role  in  the  eventual  acceptance 
of  simulation  as  a  noble  pursuit  (Schruben  2011).  Dr.  Lewis’s  impact  was  so  great,  in  fact,  that  he  was 
recently  awarded  the  2012  INFORMS  Lifetime  Professional  Achievement  Award  (LPAA)  in  recognition 
of  his  contributions. 

4  DISCUSSION 

The  previous  results  show  that  simulation  publications  in  two  of  the  search  engines  (ACM-DL  and  EB- 
SCOhost)  have  roughly  kept  pace  with  Moore’s  Law.  In  the  other  two  search  engines  (JSTOR  and 
INFORMS)  they  have  polynomial  growth  patterns  over  time.  In  addition,  deterministic  analytic  models 
are  no  longer  achieving  the  popularity  of  stochastic  models  that  are  simulation  based.  A  comparison  of 
search  engine  searches  for  the  analytic  keywords  “linear  programming”  and  “optimization”  shows  that  the 
growth  of  these  particular  topics  has  not  kept  pace  with  that  of  simulation. 

4.1  Summary  of  Exponential  Simulation  Growth  Curves 

The  four  data  sets  resulted  in  four  separate  and  distinct  models  that  were  used  to  determine  the  most  in¬ 
fluential  years  in  simulation  history.  While  some  of  the  models  share  influential  years,  they  were  other¬ 
wise  completely  different  models.  They  also  share  an  obvious  exponential  trait.  As  such,  they  each  con¬ 
tribute  some  validity  to  a  theory  offered  by  B.  W.  Hollocks  in  his  2006  paper,  “Forty  years  of  discrete- 
event  simulation — a  personal  reflection.”  Hollocks  provided  an  in-depth  look  at  technological  advances 
that  he  has  observed  throughout  his  time  in  the  OR  community.  He  suggested  a  tweak  to  Moore’s  Law  as 
it  applies  to  simulation.  Instead  of  a  trend  based  on  values  that  would  double  every  1 8  months  “until  con- 


Powers,  Sanchez,  and  Lucas 


strained  by  the  laws  of  physics,”  Hollocks  predicted  that  simulation’s  evolution  would  follow  more  of  an 
“S-curve,”  as  shown  in  Figure  5  (Hollocks  2006). 


Figure  5:  S-curve  of  simulation  growth  (Hollocks  2006). 

The  curve  proposed  by  Hollocks  says  that,  as  computing  capability  reaches  its  physical  upper- 
bound — a  bound  that  Gordon  Moore  himself  predicted  would  occur  around  2015  to  2020 — limited  scope 
for  improvement  might  be  left.  What  room  does  remain  may  be  user-oriented  improvements  such  as  ease 
of  model  building  and  display  capabilities.  Such  a  theory,  however,  would  mean  that  simulation  has  en¬ 
joyed  success  and  expansion  over  time  mainly  because  of  technological  advances.  A  “squinty-eyed”  look 
at  the  curves  presented  in  this  research  may  suggest  that  Hollocks  is  correct  in  giving  so  much  credit  to 
improvements  in  processing  speed  and  software.  Of  advanced  processing  speed,  Hollocks  said  that  “[i]n 
addition  to  permitting  the  same  tools  to  run  faster,  it  has  also  permitted  the  tools  to  carry  more  function 
and  feature,  leading  to  the  current  status  of  simulators.”  Simply  put,  Hollocks  is  suggesting  that  the  simu¬ 
lation  field  is  getting  very  close  to  being  as  advanced  as  it  will  ever  be  (Hollocks  2006).  The  authors  of 
this  paper  have  another  suggestion. 

4.2  What  is  Next  For  Simulation 

Computational  power  provides  benefits  for  both  modeling  and  analysis.  As  we  mention  in  the  analysis  of 
the  ACM  DL  data,  the  simulation  community  may  be  poised  for  breakthroughs  that  augment  enhanced 
processing  speed.  The  technological  ceiling  that  Moore  has  predicted  to  be  right  around  the  corner  may 
still  hold  true  for  the  physical  limits  of  computers  and  speed,  but  the  future  is  wide  open  for  simulation. 
At  least  it  will  be  when  improved  analysis  approaches,  such  as  simulation  optimization  and  efficient  de¬ 
sign  of  experiments,  are  readily  available  to  the  simulation  analyst  and  in  widespread  use. 

A  recent  leap  of  note  was  the  June  2012  unveiling  of  a  supercomputer  known  as  IBM’s  “Sequoia,”  a 
bank  of  over  1.5  million  processing  cores  capable  of  performing  over  16  quadrillion  operations  per  se¬ 
cond  (16  petaflop/s)  (Wait  2012).  Such  capability  is  so  difficult  to  conceptualize  that  it  is  not  unreasona¬ 
ble  to  say  that  it  may  be  approaching  the  limits  of  physics.  It  is  difficult  to  imagine  a  model  that  can  act 
more  efficiently  than  one  that  is  being  executed  on  the  type  of  technology  that  can  support  petaflop-speed 
calculation.  It  would  seem  that  such  massive  computing  power  would  be  enough  to  explore  the  limits  of 
even  the  most  complex  model.  However,  due  to  the  “curse  of  dimensionality”  even  this  level  of  computa¬ 
tional  power  is  not  sufficient  to  provide  a  “brute-force”  analysis  for,  e.g.,  conducting  thorough  sensitivity 
analyses  of  the  complex  simulation  models  that  are  pervasive  in  operations  research,  industry,  business, 
and  military  simulations.  In  fact,  a  “brute-force”  approach  involving  a  single  replication  of  a  two-level 
experiment  that  involves  100  factors  (examined  at  all  2 100  potential  combinations)  on  a  model  consisting 
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of  a  single  elementary  operation  would  take  over  2.5  million  years  to  complete.  The  moral  is  that  advanc¬ 
es  in  simulation  cannot  rely  on  advances  in  technology  alone. 

Fortunately,  we  need  not  wait  for  suitable  analysis  approaches  to  be  developed.  If  simulation  has 
started  to  level  out  in  terms  of  technology-based  advancements,  then  the  design  of  experiment  (DOE) 
methods  have  the  potential  to  break  the  ceiling.  Imagine  the  aforementioned  two-level,  1 00  factor  exper¬ 
iment  as  the  “ceiling”  for  a  machine  as  capable  as  IBM’s  Sequoia.  This  is  a  stretch  of  the  imagination,  as 
not  even  the  most  patient  analyst  can  spare  millions  of  years  for  a  single  replication.  Nonetheless,  smart 
experimental  designs  allows  an  analyst  to  identify  the  significant  main  effects  and  a  modest  number  of  in¬ 
teractions  among  these  100  factors  using  as  few  as  a  few  hundred  carefully-specified  combinations.  For 
example,  thanks  to  the  DOE  method  of  fractional  factorials,  commonly  known  as  “screening  designs”  for 
identifying  important  factors  from  an  experiment,  there  exists  a  resolution  V  2120'105  design  for  two-level 
experiments  consisting  of  up  to  120  factors  that  allows  for  all  main  effects  and  two-way  interactions  to  be 
estimated  in  32,768  runs,  and  this  can  easily  be  expanded  into  a  design  permitting  the  estimation  of  full 
second-order  metamodels  (Sanchez  and  Sanchez  2005). 

Similarly,  imagine  a  simulation  model  running  on  a  computer  with  processing  speed  that  allows  for 
each  design  point  to  take  only  one  second  to  run.  Such  a  scenario  is  often  feasible  considering  the  pro¬ 
cessing  speeds  of  commercially  available  systems.  Now  suppose  that  this  model  is  comprised  of  29  fac¬ 
tors.  The  “brute  force”  simulation  approach  would  require  over  1 7  years  using  a  229  full  factorial  design. 
This  takes  the  wind  out  of  the  ability  to  simply  ride  the  technological  wave.  Now,  apply  the  benefits  of 
Latin  hypercube  sampling  and  the  same  model  would  take  under  five  minutes  using  a  257  design  point 
NOLH  design  (Cioppa  and  Lucas  2007).  In  other  words,  one  can  efficiently  squeeze  a  substantial  amount 
of  experimental  information  from  such  a  29-factor  experiment  in  less  than  five  minutes.  Efficient  designs 
also  allow  for  flexibility  in  modeling,  which  is  particularly  useful  in  a  world  where  the  analyst  must  ac¬ 
count  for  last-minute  “good  ideas”  offered  by  his  or  her  clients  and  there  may  be  considerable  uncertainty 
about  the  significant  factors  and  the  response  forms. 

The  theories  behind  DOE  cannot  take  full  credit  for  such  flexibility.  It  is  a  combination  of  the  com¬ 
putational  capability  available  on  modem  computers  and  the  “art”  of  experimental  design.  This  marriage 
of  the  processing  power  of  today’s  machines  and  the  power  provided  by  efficient  experimental  design  is 
an  example  of  both  sides  of  the  simulation  argument  playing  nice.  The  idea  of  optimality  is  under  the 
hood  of  many  efficient  designs  in  that  the  combination  of  factor  settings  provides  the  “best”  estimation  of 
a  specified  response.  Furthermore,  the  concept  of  “optimality”  permeates  the  practical  real  world  where 
“sampling  is  expensive — the  goal  is  to  take  no  more  samples  than  absolutely  necessary.”  That  is,  even  if 
you  could  spare  the  time  to  run  as  many  replications  of  your  complex  model  as  you  wanted,  you  do  not 
have  to  (Sanchez  and  Wan  2012).  The  optimality  of  efficient  design  is  holding  hands  with  the  impressive 
capabilities  of  modern  computers  to  create  simulation  models  that  represent  a  powerful  force  that  is  any¬ 
thing  but  “brute.”  Of  course,  another  advantage  of  many  space-filling  designs,  such  as  NOLHs,  is  that 
they  are  robust  in  the  sense  that  they  allow  analysts  to  explore  a  breadth  of  diverse  possible  metamodels 
(Kleijnen  et  al.  2005;  Vieira  Jr.  et  al.  2011).  For  more  on  the  use  of  experimental  design  for  simulation 
(including  additional  references),  please  see  the  tutorial  by  Sanchez  and  Wan  (2012)  earlier  in  this  Pro¬ 
ceedings.  Advances  in  simulation  optimization  (Fu  2002)  are  also  providing  new  opportunities  for  simu¬ 
lation  analysts;  there  are  several  sessions  in  the  2012  WSC  Proceedings,  and  an  entire  track  in  the  2011 
WSC  Proceedings. 

In  order  for  analysts  to  take  advantage  of  new  design  and  analysis  capabilities,  they  need  the  proper 
software  tools.  Fortunately,  new  simulation  environments  are  making  it  easier  to  build  simulations,  con¬ 
duct  many  experiments  in  parallel  using  efficient  experimental  designs,  and  analyze  the  output  with  statis¬ 
tical  tools  and  graphical  playbacks.  The  future  of  simulation  is  moving  beyond  the  emphasis  on  program¬ 
ing  and  conducting  a  small  number  of  experiments.  We  see  a  future  in  which  modern  design  and  analysis 
capabilities  allow  researchers  to  get  orders  of  magnitude  more  experimental  information  from  their  mod¬ 
els — and  that  these  possibilities  become  the  standard  practice.  Indeed,  the  simulation  community  is  just 
beginning  to  utilize  these  powerful  capabilities. 
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5  CONCLUSIONS 

The  exponential  nature  of  simulation’s  growth  over  time  has  been  made  apparent  without  a  need  for  deep 
exploration  or  analysis.  The  model  fitting  and  analysis  of  influential  years  has  pointed  out  some  possible 
candidates  for  the  most  significant  advances  in  the  computing  and  academic  fields.  The  takeaway  is  that 
simulation  did  not  simply  ride  the  technological  wave  and  only  take  advantage  of  improved  processing 
speeds.  There  have  been  serious  contributions  in  terms  of  elegant  thought,  such  as  new  algorithms  for 
pseudo-random  number  generation,  simulation  optimization,  and  efficient  design.  As  we  educate  the  next 
generation  of  simulation  practitioners  and  researchers,  it  is  important  to  let  them  know  that  the  “last  re¬ 
sort”  stigma  that  has  been  so  aggressively  applied  to  simulation  in  the  past  is  not  an  accurate  reflection  of 
the  field.  Furthermore,  as  technologies  that  leverage  higher  processing  capabilities,  such  as  simulation 
software  and  efficient  high-dimensional  design  of  experiments,  continue  to  advance,  we  expect  the  expo¬ 
nential  growth  in  the  use  of  simulation  to  continue  for  the  foreseeable  future.  For  many  complex  real 
world  problems  requiring  complex  models,  simulation  will  increasingly  be  the  tool  of  “first  resort.” 
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